System and method for wireless network performance measurement and management using remote devices

ABSTRACT

A system for measuring and reporting wireless network service quality using remote devices, has been devised. The system comprises a central analysis and control server comprising at least a plurality of programming instructions stored in a memory and operating on a processor of a network-connected computing device; a mobile wireless test device comprising at least a plurality of programming instructions stored in a memory and operating on a processor of a network-connected computing device, which may be an application on an end-user device or a dedicated device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/161,177, titled “SYSTEM AND METHOD FOR WIRELESS NETWORK PERFORMANCEMEASUREMENT AND MANAGEMENT USING REMOTE DEVICES” filed on May 20, 2016,which is a continuation of U.S. patent application Ser. No. 15/161,172,titled “SYSTEM AND METHOD FOR WIRELESS NETWORK PERFORMANCE MEASUREMENTAND MANAGEMENT USING REMOTE DEVICES”, filed on May 20, 2016, now issuedas U.S. Pat. No. 10,251,120 on Apr. 2, 2019, the entire specification ofwhich is incorporated herein by reference in its entirety.

BACKGROUND OF THE INVENTION Field of the Art

The disclosure relates to the field of electronics networking, and moreparticularly to the field of wireless network service qualitymeasurement and management.

Discussion of the State of the Art

All must recognize that the advent and later widespread use of computernetworking has revolutionized the operation of virtually all businessinstallations from smallest outlet to warehouse, factory and corporateheadquarters. At first it was slow speed wired service which wassomewhat constricting, difficult to set up and required the laying ofpossibly miles of sometimes very inconvenient wiring but speeds improvedrapidly and most corporations have become fully dependent on networkingand cloud access to function. More recently, the availability, betterspeed and increased reliability of wireless networking has once againbegun large scale changes in corporate operations. Now one may havecontinuous network connectivity with all of the corporate resources andthe cloud, but may now move around the corporate landscape at willdepending on the mobility of the device being used. Device mobility hasgreatly changed at the same time. While wireless networking is indeed alarge step forward, it does not come without drawbacks as networks maybecome over-subscribed without much recognizable warning, one does notneed to worry that there are enough ports on the switch to accommodateeveryone, but the capacity constraint is still there. Also wirelesscoverage is greatly affected by environmental factors, buildinginfrastructure such as walls and beams, radio interference eitherchronic from other office equipment or items employees bring ortransient cloud cover, sun spots, humidity, etc. Coverage at peripheralareas of a building complex so that all employees have fast optimalservice without “leaking” your network outside of the corporate campusto where it is a beacon for hacking are all important considerations.There are many tests that may be run that probe an aspect of wirelessnetwork service delivery and quality, but knowing which ones to run, howoften to run them, from where to run them, and how to interpret theresults, are daunting factors. Budgeting such endeavors is also no smallconsideration.

What is needed is a system and method that will comprehensively test acompany's wireless network equipment and coverage, retrieve data fromall available equipment, coordinate third party parts of the testsystem, automatically run important tests without intervention, use enduser devices or testing equipment that is already or may easily beplaced throughout the coverage map of the tested network and willtransform and then display the data in such a way that all concernedwith network health and function may understand and show to those incharge of the purse strings to resolve issues may easily comprehend, forexample graphs and minimal service levels, possibly with remedysuggestions when something does go wrong.

SUMMARY OF THE INVENTION

Accordingly, the inventor has conceived and reduced to practice, in apreferred embodiment of the invention, a system and method for wirelessnetwork performance measurement and management using remote devices.

To provide a means to measure wireless network service quality from aplurality of locations and access points within that network, theinventors conceived and reduced to practice a system where wirelessnetwork connected mobile devices such as smart phones and tablets may beused in conjunction with a central data analysis and test control serverto measure multiple parameters that reflect wireless network servicequality. To participate, a given mobile device downloads and installs ageneric test suite application and enters an identifying code which hasbeen provided separately. Device sends this code to the central dataanalysis and test control server. The central server responds with thecoding and configuration for tests to be run. These tests may includewireless signal strength, wireless current channel, wireless frequencyband, data throughput, network capacity currently in use, radioattachment latency and success, ping latency, remote resource accesslatency and remote resource download bandwidth among many otherparameters possible. Test parameters define details of the tests,including networks to be tested and target servers to be used in thetest. Test parameters also determine whether tests are run periodically,continuously or initiated by user only. Tests performed by the testsuite may be chosen to reflect the needs and purpose of individualclients or test campaign at the central data analysis and test controlserver and then pushed to the remote devices just prior to the start oftesting. The identification code allows customizing a standard device ora group of standard devices with a generally available software for thepurpose of the customized test. The identification code is alsosubmitted with the test data points and allows storing and analyzing thedata centrally for a group of devices which belong to certainorganization.

The wireless network service quality test system may be configured toreceive information from third party through an application programminginterface “API”. This data may include data from devices such aswireless access points to receive information that may include themanufacturer and model of the access point, map of the area, accesspoint location in the map, GPS coordinates of the access points,location of certain device at certain time as positioned by the network,current attached clients, current error logs and current firmware andsoftware level, among other information. This information may be used tobetter complete the dataset both during routine testing and in casetroubleshooting analytics are needed.

The API may also be used to send information such as warnings, alarmsand wireless network performance data to third party devices such asgeneral network managers, hospital equipment and supervisory devicesdependent on effected wireless networking or mobile devices of networkadministrative personnel. These data or messages may be routine, orinformative or may signify that service level of a particular aspect oraspects of a monitored wireless network has dropped below apre-designated minimal level required by the customer. Alternatively theAPI may provide all raw measurement data to other applications forfurther processing. Analytics capability of the wireless network servicequality test system may also be used to predict the root cause of theservice reduction and suggest possible remedial action, if desired.

Finally, data collected by the remote service quality test devices thatare part of the system and analyzed by the central data analysis andtest control server may be represented and displayed to best suit thecustomer's needs. Some examples are showing one or more selected servicequality parameters graphically over a pre-selected time period as apercentage of known service target level of the tested network, possiblyalso depicting pre-decided minimal service levels. Another possibilitymay be a color-coded topographical type map of the coverage area of thetested network showing signal levels or network throughput as differingcolors as a function of location. Yet another possibility is to displayadherence to pre-determined service levels with colored cells on a mapor floor plan. This would make use of location reporting functions ofthe test devices, possibly GPS based, or location data gleaned from thewireless access points or other electromagnetic signal transmittersbased on signal levels. These may use radio frequencies, microwavefrequencies or even optical light. As well, propagation time andcomparative target signal strength of the test device may be utilized todetermine the location. Location may also be entered by user of thedevice by pointing out a location on a map or floor plan. Many otherrepresentations are possible, dependent on the needs and goals of thecustomer. Data may be encoded in such a way so as to be used by anotherelectronic system such as a third party network manager.

According to a preferred embodiment of the invention, A system forwireless network performance measurement and management using remotedevices, comprising: a central analysis and control server comprising atleast a plurality of programming instructions stored in a memory andoperating on a processor of a network-connected computing device, amobile wireless device comprising at least a processor and a memory, anda wireless network testing software application stored in the memory andoperating on the processor of the wireless device. The central analysisand control server: transmits wireless network testing softwareapplication to at least one wireless device, receives result data ofwireless network service quality testing performed by at least onewireless device, and analyzes test data of wireless network servicequality testing using pre-determined transformations to demonstrate keyaspects of service quality; and represents transformed result data asbest suited for further steps towards overall wireless service qualitymanagement. The mobile wireless device: comprises a plurality of mobilewireless device types routinely operated by a plurality of customer'semployees, performs sets of wireless network service quality tests dueto the downloaded wireless network testing software application, iscapable of performing tests in the background while a user performsother conventional wireless device related functions; and transmitswireless network service quality test data to central analysis andcontrol server. The wireless network testing software application:programs mobile wireless device to carry out pre-designed, directedactive examination of wireless network functions and passive monitoringof network parameters to establish level of wireless network servicequality, and allows display of wireless network service quality tests onthe wireless mobile device executing it.

According to another embodiment of the invention, at least one remotewireless test device is a network enabled mobile device such as asmartphone or tablet running a specifically programmed wireless networktest suite framework as an application. Remote wireless test devices maybe wireless network enabled laptop computers, desktop computers, cablemodems, ADSL routers, Wi-Fi access points, mobile network base stations,network switches, plug in devices (USB, or similar) in addition to thosementioned. A large plurality of remote wireless test devices may beemployed during network service quality monitoring of a single wirelessnetwork. Dedicated network target devices may be used to provideadditional operational data as part of wireless network service qualitytests. Information from one or more third party network infrastructuredevices may be integrated into wireless network service quality testresults to provide additional operational information. Data collected byremote wireless test device and analysis functions employed by centralanalysis and control server may differ based upon a client'sspecifications. The wireless network service quality parameterscollected by remote wireless test device come from a set of wirelessnetwork parameters that include: wireless network signal strength,wireless network channel, wireless network signal frequency band,wireless network standard in use, connected wireless access point,network radio attachment time and success rate, network data throughput,voice quality, web page load time and success rate and ping latency topre-selected target, RF characteristics of the test signal like retryrates and data rates among other parameters known to the art. Thecentral analysis and control server may accept and store wirelessnetwork user comments concerning service quality and third partyapplication service quality data.

According to another embodiment of the invention, the wireless networktesting software application tests programmed occur on a repetitiveperiodic basis wake sleeping wireless device, an active test is loadinga web page or logging into a certain site active tests like throughputand voice quality target custom endpoints, the transmission of testresults are delayed with result data stored locally if connectivity tothe central analysis and control server is unavailable, results aresubmitted when connection is restored, additional network configurationdata is imported from network equipment and then correlated with testdata at central analysis and control server, service level agreementtarget thresholds are used to determine sufficient wireless networkservice quality, while one network is used by the wireless device, asecond network may be tested in the background, test configurationparameters define which networks are tested, if wireless device cannotconnect to one or more of the defined networks, absence of networkconnection is recorded and the active tests deferred.

According to another embodiment of the invention, a system for wirelessnetwork performance measurement and management using remote devices,comprising customer specific test customizations such as correct testprofiles and SLA thresholds are downloaded from a central server to themobile wireless device when the user enters a code. Customer specifictest customizations such as correct test profiles and SLA thresholds aredownloaded from a central server to the mobile wireless device when theuser enters pre-existing profile credentials which causes the user to beadded to a specific organization for data aggregation. Passive usernetwork traffic monitoring, observed network responsiveness and networkaccess data collected while user uses the mobile wireless device. Testsinclude passive measurement for collection wireless radio signal relatedand network traffic load data. Active tests are postponed when user runstasks on the mobile wireless device that result in network usage or CPUload above pre-set threshold. At least a subset of test data fromindividual mobile wireless devices is be correlated with similarwireless mobile devices in a central database. User feedback concerningwireless network service quality in integrated within the wirelessnetwork testing software application, user feedback collection triggeredat discretion of an administrator of the central analysis and controlserver or by pre-determined SLA violations determined by the mobilewireless device application. At least one data set which includeslocation information is layered over a scalable map. A pre-programmedroaming test monitors mobile wireless device connections and transitionsbetween cells and networks.

According to another embodiment of the invention, A method for wirelessnetwork performance measurement and management using remote devices,comprising the steps of: a) receiving a wireless network testingsoftware application encoding pre-designed wireless network testfunctions using a wireless device from a central analysis and controlserver after entering an identity code; b) employing the wirelessnetwork test functions to both actively probe and passively monitorparameters impacting wireless network service quality level usingmonitoring devices, at least in part, consumer level mobile wirelessdevices; c) transmitting data resultant from wireless network testfunctions to the central analysis and control server; d) transformingwireless network test function data using central analysis and controlserver and formatting it for display or further action as predeterminedby test administrators

BRIEF DESCRIPTION OF THE DRAWING FIGURES

The accompanying drawings illustrate several embodiments of theinvention and, together with the description, serve to explain theprinciples of the invention according to the embodiments. It will beappreciated by one skilled in the art that the particular embodimentsillustrated in the drawings are merely exemplary, and are not to beconsidered as limiting of the scope of the invention or the claimsherein in any way.

FIG. 1 is a block diagram illustrating an exemplary system architecturefor measuring and reporting wireless network service quality usingremote devices, according to a preferred embodiment of the invention.

FIG. 2 is a flow diagram illustrating an exemplary method for measuringand reporting wireless network service quality using remote devices,according to a preferred embodiment of the invention.

FIG. 3 is a flow diagram illustrating an exemplary method by which thesystem is programmed with customer specific test suites according to apreferred embodiment of the invention.

FIG. 4 is a block diagram illustrating an exemplary system architecturefor central analysis and control server to analyze network servicequality using remote devices, according to a preferred embodiment of theinvention.

FIG. 5 is a flow diagram illustrating an exemplary method by whichcentral analysis and control server may interact with other componentsof the system and analyze network performance test data according to apreferred embodiment of the invention.

FIG. 6 is a diagram illustrating information that is processed bycentral analysis and control server analytics engine according to apreferred embodiment of the invention.

FIG. 7 is a block diagram illustrating an exemplary system architecturefor mobile wireless service quality test device employed to analyzenetwork service quality using remote devices, according to a preferredembodiment of the invention.

FIG. 8 is a flow diagram illustrating an exemplary method by whichmobile wireless network service quality device may operate to collectstore and transmit test suite data according to a preferred embodimentof the invention.

FIG. 9 is a diagram illustrating methods by which users of wirelessnetworks being tested might submit their perceived experience to thewireless network service quality test system according to a preferredembodiment of the invention.

FIG. 10 is a diagram illustrating both passive and active testing ofmore than one wireless network by wireless network service quality testdevices according to a preferred embodiment of the invention.

FIG. 11 Is a list of methods by which the location of individualwireless network service quality test devices may be obtained accordingto a preferred embodiment of the invention.

FIG. 12 is a diagram displaying two example formats of wireless servicequality test output according to a preferred embodiment of theinvention.

FIG. 13 is a flow diagram of a possible notification, warning and alarmmechanism to be used when network performance is found to be belowcustomer established service level minimums according to a preferredembodiment of the invention.

FIG. 14 is a block diagram illustrating an exemplary hardwarearchitecture of a computing device used in an embodiment of theinvention.

FIG. 15 is a block diagram illustrating an exemplary logicalarchitecture for a client device, according to an embodiment of theinvention.

FIG. 16 is a block diagram showing an exemplary architecturalarrangement of clients, servers, and external services, according to anembodiment of the invention.

FIG. 17 is another block diagram illustrating an exemplary hardwarearchitecture of a computing device used in a plurality of embodiments ofthe invention.

FIG. 18 is a diagram illustrating change of access point during wirelessnetwork user roaming according to an embodiment of the invention.

FIG. 19 is a table of network parameters that may be tested according toan embodiment of the invention.

DETAILED DESCRIPTION

The inventor has conceived, and reduced to practice, in a preferredembodiment of the invention, a system and method for wireless networkperformance measurement and management using remote devices.

One or more different inventions may be described in the presentapplication. Further, for one or more of the inventions describedherein, numerous alternative embodiments may be described; it should beappreciated that these are presented for illustrative purposes only andare not limiting of the inventions contained herein or the claimspresented herein in any way. One or more of the inventions may be widelyapplicable to numerous embodiments, as may be readily apparent from thedisclosure. In general, embodiments are described in sufficient detailto enable those skilled in the art to practice one or more of theinventions, and it should be appreciated that other embodiments may beutilized and that structural, logical, software, electrical and otherchanges may be made without departing from the scope of the particularinventions. Accordingly, one skilled in the art will recognize that oneor more of the inventions may be practiced with various modificationsand alterations. Particular features of one or more of the inventionsdescribed herein may be described with reference to one or moreparticular embodiments or figures that form a part of the presentdisclosure, and in which are shown, by way of illustration, specificembodiments of one or more of the inventions. It should be appreciated,however, that such features are not limited to usage in the one or moreparticular embodiments or figures with reference to which they aredescribed. The present disclosure is neither a literal description ofall embodiments of one or more of the inventions nor a listing offeatures of one or more of the inventions that must be present in allembodiments.

Headings of sections provided in this patent application and the titleof this patent application are for convenience only, and are not to betaken as limiting the disclosure in any way.

Devices that are in communication with each other need not be incontinuous communication with each other, unless expressly specifiedotherwise. In addition, devices that are in communication with eachother may communicate directly or indirectly through one or morecommunication means or intermediaries, logical or physical.

A description of an embodiment with several components in communicationwith each other does not imply that all such components are required. Tothe contrary, a variety of optional components may be described toillustrate a wide variety of possible embodiments of one or more of theinventions and in order to more fully illustrate one or more aspects ofthe inventions. Similarly, although process steps, method steps,algorithms or the like may be described in a sequential order, suchprocesses, methods and algorithms may generally be configured to work inalternate orders, unless specifically stated to the contrary. In otherwords, any sequence or order of steps that may be described in thispatent application does not, in and of itself, indicate a requirementthat the steps be performed in that order. The steps of describedprocesses may be performed in any order practical. Further, some stepsmay be performed simultaneously despite being described or implied asoccurring non-simultaneously (e.g., because one step is described afterthe other step). Moreover, the illustration of a process by itsdepiction in a drawing does not imply that the illustrated process isexclusive of other variations and modifications thereto, does not implythat the illustrated process or any of its steps are necessary to one ormore of the invention(s), and does not imply that the illustratedprocess is preferred. Also, steps are generally described once perembodiment, but this does not mean they must occur once, or that theymay only occur once each time a process, method, or algorithm is carriedout or executed. Some steps may be omitted in some embodiments or someoccurrences, or some steps may be executed more than once in a givenembodiment or occurrence.

When a single device or article is described herein, it will be readilyapparent that more than one device or article may be used in place of asingle device or article. Similarly, where more than one device orarticle is described herein, it will be readily apparent that a singledevice or article may be used in place of the more than one device orarticle.

The functionality or the features of a device may be alternativelyembodied by one or more other devices that are not explicitly describedas having such functionality or features. Thus, other embodiments of oneor more of the inventions need not include the device itself.

Techniques and mechanisms described or referenced herein will sometimesbe described in singular form for clarity. However, it should beappreciated that particular embodiments may include multiple iterationsof a technique or multiple instantiations of a mechanism unless notedotherwise. Process descriptions or blocks in figures should beunderstood as representing modules, segments, or portions of code whichinclude one or more executable instructions for implementing specificlogical functions or steps in the process. Alternate implementations areincluded within the scope of embodiments of the present invention inwhich, for example, functions may be executed out of order from thatshown or discussed, including substantially concurrently or in reverseorder, depending on the functionality involved, as would be understoodby those having ordinary skill in the art.

Conceptual Architecture

FIG. 1 is a block diagram illustrating an exemplary system architecture100 for measuring and reporting wireless network service quality usingremote test devices 115, according to a preferred embodiment of theinvention. Under the embodiment, data concerning factors that affect theservice quality levels of a wireless network of interest 120 which mayinclude but are not limited to steady state signal strength, variabilityin signal strength, signal to noise ratio, packet retry rate, used datarate, current percentage of network traffic versus network capacity,radio attachment latency and success rate, resource request latency, webpage load time and success rate, access to certain applications overnetwork connection, voice quality, video quality, packet loss, jitter,location of the test device, and ping response time, among others knownto those skilled in the art, are collected by wireless network connectedremote test device 115 which most often will be an end-user's mobiledevice such as a smart phone or tablet running a pre-programmed networktest framework application. Remote network test device 115, may also,under some circumstances, be a desktop workstation, a laptop computer, akiosk, an Internet of Things (IoT) device, a wireless access point,modems with wireless (“Wi-Fi”) or optical (“Li-Fi”) capability such asasynchronous digital subscriber link (“ADSL”) or cable modems, ordedicated sensor devices for this purpose. Other test devicepossibilities exist; those put forth here are meant only as selectedexamples. A plurality of remote network test devices 115 may be used ineach network service quality test.

While simple spot tests of a network are possible using only the remotenetwork test device with the un-programmed network test frameworkapplication, the data collected in this fashion is not associated withdata collected from concurrent testing run by other mobile devices andthe data are not stored in such a way as to be later used to diagnosenetwork issues or render time based status results of a network. To runcontinuous or recurring ongoing tests that may be used in furtheranalysis and conclusion generation, the client must have an accountwhich holds, among other information the specific network tests to run,the duration of the data collection for each test, where appropriate,the periodicity at which each test should be run, the number ofrecurrences for each test, the transformations to be run on eachreturned data set and potentially, the manner in which results are to bepresented, to name a few examples of parameters that may comprise a testfrom the many known within the field. In this embodiment all of theseparameters, with the programming functions to execute the desirednetwork evaluation are stored in a cloud-based test code andconfiguration server 116 and, prior to testing, information, including aunique test suite ID, is transferred to a site's central analysis andcontrol server 111 from which remote network test devices 115 to be usedare programmed and configured. All data collected as part of the testsuite will have the unique test suite ID when sent to central analysisand control server 111 and may be stored in a data store 113 long-termas such including the unique ID for retrieval and future transformationby central analysis and control server 111. Central analysis and controlserver 111 also provides an API 112 that allows it to share output fromtest analysis to third party network equipment 124 such as a networkmanagement server, to name one of a plurality of examples, and toretrieve important information from wireless network equipment such asmaps with access point locations, wireless network firmware and softwareversions, network configuration information, traffic volume and qualityparameters from network equipment, wireless access point 121 parameterswhere such data as wireless standards being supported (examples: b, g,a, n, ac), security protocols enforced (examples: WPA, WPA2, WPAEnterprise), access point RF radio transmitted signal strength,broadcast bands enabled (examples: 2.4 GHz, 5 GHz), collisions,retransmissions and percent capacity in use, to list a few members of alarger set of attributes, may be important to diagnosing wirelessnetwork service quality issues, proposing remedies, and sendingnotifications, capabilities for which the analytics engine of centralanalysis and control server 111 is programmed should network servicelevels fall below customer service level expectations.

Based upon programming and test system setup, the remote network testdevices may attempt to connect to and download or upload resources tocompany internal dedicated target test devices 122. These devices may beconnected to the same networked by cable or connected wirelessly. Theymay also be connected to another segment or network within the companyto allow testing of a wider range of infrastructure components. Theremay also be other devices present on a company's network infrastructurethat are critical to operations that are therefore tested 123. Anexample may be wired or wirelessly connected printers that the companyuses which, if not available for jobs, may cause process delays or workstoppages; some may print manufacturing orders, others invoices andstill others general duty jobs. Another example may be wirelesslyconnected or wired scanners. Service requirements for these resourcesare expected to be high and action in event of significant reduction orloss of function needed swiftly. Similarly, remote devices performingtest may be programmed to automatically log in to certain applicationslike warehouse control systems or sales management systems to testavailability of access to them and possibly perform a test query torecord overall service availability and response times. Measurement ofservice to external cloud 130 resources 131, 132, 133 is equallyimportant and testing connectivity and resource availability fromexternal dedicated targets 131, discretionary targets 133, perhaps agovernment printer or software as service systems that the company mightuse. The ability to reach a wide number of high volume web server webpages 132, for example FACEBOOK™, SKYPE™, CHROME™ and GOOGLE™ to list afew known to those skilled in the art, as well as customer critical webserver web pages is also a good indicator of wireless network servicehealth as it relates to traversing firewalls and gateways, and thehealth of web servers of specific concern to the customer, if included.

Devices accessing network use several services which control access toresources and provide basic connectivity information. DHCP servicesallocate IP addresses to devices connecting to network. Authenticationand authorization services like Radius provide a wide variety ofprotocols for authenticating the users prior to allowing access toresources. DNS services allowing use of URLs instead of numeric IPaddresses. Device with the test suite use and tests these services andreport qualities like success rates, delays and errors with theservices. This information is stored and analyzed for further actions.

Devices programmed for test may perform measurements in the backgroundmeasuring signal levels, data rates, retry rates and throughputs atdifferent times. While the device moves, signal levels vary or wheninstructed by network, device may change connection to another accesspoint as illustrated in FIG. 18. Assessing characteristics of theroaming behavior can be done be measuring signal level, signal to noiseratio, used data rate, throughput, packet loss and other parametersbefore and after the roaming event 1800. Measuring the time gap betweenthe last data packet from previous access point and the first packetfrom the new access point provides information on the packet flowinterruptions, which are especially important for real time traffic likevoice. Collecting this information allows comparing behavior ofdifferent device manufacturers, device models, software versions, accesspoints and different network settings and optimizing the service forbest user experience.

Resolving certain issues may require more detailed information than thenormal test suite can provide. Central analysis and control server mayinitiate a more detailed test procedure to selected devices. This mayinclude observing and measuring terminal behavior continuously,collecting logs from the terminal, collecting logs triggered by failureto meet predetermined service parameters, entering a special test modeor root/jail break mode which provides more information or recording thedevice display views at the time of the issue. Central control andanalysis server may also be used to ask user to perform certain task.

Test controls define when tests are performed. Test controls include alist of networks which are to be tested. In the absence of an includednetwork, no active test will be attempted. This allows collection ofdata only from networks of interest, minimized terminal batteryconsumption and bandwidth consumption if network is metered.

The system also has capability to accept and analyze user experienceconcerning wireless network service quality during testing 115. Eitheras a matter of normal test suite function or due to the return of testresults showing borderline or deficient service quality levels, usersmay be asked to comment either being given multiple preprogrammedanswers from which to choose or through freeform written response whichis interpreted by the central analysis and control server 111. Extradata provided by receiving user experience feedback provides a morecomplete picture of impact of lower service levels on actual userexperience.

It should be noted that placement components of the invention in thisembodiment were made for explanatory reasons. Some components, forexample the central analysis and control server, could reside elsewherewithout changing its function. The central analysis and control serverwould function similarly as a corporation local device or as a clouddevice, possibly a software as a service appliance subscribed to by theclient corporation.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

FIG. 2 is a flow diagram illustrating an exemplary method 200 formeasuring and reporting wireless network service quality using remotedevices, according to a preferred embodiment of the invention. Toperform reproducible testing at least one, probably more than one, ofremote wireless network testing devices 115 as depicted in FIG. 1 mustbe programmed with the suite of tests to be run 201 as depicted in FIG.1 116, 111, 115. These remote test devices 115 then run the testsspecified in test suite 202, which are attached to a unique test suitespecific ID, such as but not limited to: wireless signal strength attest device location; wireless network BSSID; ability and latencylogging onto the wireless network; negotiated wireless standard; signallevel, wireless signal band or bands offered; ping round trip times;ability to upload and download data to LAN resident and cloud baseddedicated targets, if available, measuring latency, data bandwidth,voice quality, video quality, packet loss, jitter; ability to accessspecified web pages and retrieve resources; ability to access thecompany's wireless phone carrier, if applicable; and ability to contactdedicated or critical phone targets and upload or download either voiceor other data based on the needs of the customer. There are of courseother tests known to the art and no feature precludes the invention fromperforming any of them should a customer desire. While there arecircumstances where the results of one or more wireless network testsare stored on the remote wireless network test device such as inabilityto functionally access the target wireless network or tests in theprogrammed test suite that specify that a second RF radio based wirelessnetwork be probed where only one RF radio exists on the test device,where, in both cases results are stored and transmitted, each associatedwith the test suite's ID upon re-establishment of connection with thecentral analysis and control server 111 as depicted in FIG. 1 routinely;however, test result data are transmitted to central analysis andcontrol server 111 with the test suite specific ID at time of testcompletion 203 such that interim results are as up-to-data as possible.At the central analysis and control server, test result data from remotewireless service quality test devices 115 may be combined withinformation retrieved from third party sources 204 such as wirelessaccess points 121, 141, that serve the tested wireless networks 120,140. Examples of information provided by wireless access points may be,but is not limited to: RF Radio transmit strength, BSSID of the accesspoint, wireless standards supported, wireless standards (example: Wi-Fib, g, a, n, ac) in use as well as device or devices using each standard,percent bandwidth capacity under which the access point is currentlyoperation and any recent warnings or failures in the access point'slogs. Information may also be retrieved from other third party devices124 as available. Central analysis and control server 111 may alsoreceive user perception data 205 concerning the function of the wirelessnetwork during testing, either through pre-designed multiple choicequestionnaires or interpreted freeform text descriptions, this isespecially probable if service deficiencies are determined by othertests in the suite as knowing user experience during those instances maybe especially helpful. Within central analysis and control server 111,data received from previously mentioned sources and possibly othersources specific to the needs or the test suite customer, aretransformed using pre-programmed analysis functions to produce arepresentation of all tested aspects of network function. While theserepresentations may be purely numerical, such a display is verydifficult to quickly analyze and results are thus usually depicted as aproportion of 100% function with customer-decided minimum service levelsas line graphs over a time period of interest 206. Given possession offloorplans and wireless access point maps, the system may also displayservice quality, coverage, congestion, failures, errors, user feedbackand similar factors as colored topographical like maps to makeinterpreting the data and possible problem areas as easy as possible.Drops in service quality below certain limits may have profoundconsequences on customer operations and therefore central analysis andcontrol server 111 has APIs to communicate with third party alarm andcontrol systems 207. For example, over the past decade the proportion ofhospital monitors and actuators that are wirelessly connected hasgreatly increased due to obvious convenience. However, if wirelessservice quality falls below a certain level within parts or all of thehospital for some reason, other methods for connect and control of thatequipment is needed. The API of the central analysis and control servermay communicate with the hospital's administration system to issue theneeded advisories and alarms per hospital protocol. The API would, ofcourse, function similarly in many other less drastic situations, butthe server also includes analytics programming that allows it to combinedata from different sources and different test suite which may uncoversmall irregularities in previous service levels or current equipmentfunction such that a specific root cause and remedy for an issue at handmay be proposed by the central analysis and control module with the data206. It may be that at least one test in the test suite is meant torecur a certain number of time, possibly indefinitely. The recurrence oftests and their time of run are controlled within the programming 208and tests will continue to repeat automatically for the lengthprogrammed.

FIG. 3 is a flow diagram illustrating an exemplary method 300 by whichthe system may programmed with customer specific test suites accordingto a preferred embodiment of the invention. As previously mentioned, itis possible for an unregistered system user to download the wirelessnetwork service quality framework application 301 onto a compatibleremote device (see FIG. 1, 115) and run basic network service qualitytests 302. While the results of the tests may be individually reviewedon the remote device 302, it may not be uploaded to a central analysisserver (see FIG. 1, 111) for further analysis or combined with othertests that might be run at approximately the same time either on thesame remote test device or on another remote test device. Nor may theresults be saved for later specific retrieval 303, and are therefore,under these conditions only and isolated snapshot of network function.If a user wants to take full advantage of the capabilities of the system304 she must sign up for service, in this embodiment the user signs upfor 100 remote test device service 305. At this point an account for theuser is created on the central server, a unique ID code is created forher and she is consulted concerning the specific tests that would bemost useful under her current wireless network operating conditions andperceived issues 306. This results in a suite of tests being created forthe user connected to her unique account ID. When prepared to starttesting the user need only enter her unique account ID into the runningframework application running on each of her remote network test devices307, which will result in her pre-programmed test suite being downloaded308 and run 310 on each of her remote network test devices and thisdata, associated with the unique account ID may be stored 309 and usedwith data bearing the same ID to perform in depth, coordinated analysisby the central server 312. With the unique ID, it is now also possibleto create more complex testing protocols that are run on each remotenetwork testing device associated with the user's ID 311, single testsmay be run, continuous tests may be run, tests may be run periodicallyin the background without intervention, or tests may be uploaded andthen triggered by central server. Finally, as the testing runs mature atthe user's location, the tests may be modified 313 to better reflect theuser's needs and these revised tests seamlessly uploaded to the remotenetwork test devices from the central server 314 without remote testdevice operator intervention.

FIG. 4 is a block diagram illustrating an exemplary system architecturefor central analysis and control server to analyze network servicequality using remote devices, according to a preferred embodiment of theinvention. As seen, 400 central analysis and control server 111, and 410has many components expected for a computing device FIG. 5, 20 such as,but not limited to: operating system 420, memory 430, CPU 455 I/Ocontroller module 465, video card 470, storage drive 495 and wirednetwork access module 475. In addition to these components, the centralanalysis and control drive also has a wireless service quality testcontrol module 450, which receives customer specified test suites andmodification parameters from a centralized possibly cloud based wirelessnetwork service quality test coding and parameter data store 490 thenuses that programming to configure a plurality of mobile wirelessnetwork service quality test devices 115, which run the test suites andcollect the correct, customer-requested data. Wireless networkperformance test data are sent back to the wireless service quality testtransform module 440 of central analysis and control server 410 wherespecific preprogrammed algorithms are performed against them toestablish performance levels using multiple aspects of wireless networkoperations additional data received from third party sources, such aswireless access points within the probed wireless network and receivedthrough application programming interface (API) module 460 may also takepart in the calculations. The analytics engine module 445 may also takepart in the routine calculations. Results of the test suite may be shownto end users on system interaction display 480, which may range from adedicated display and keyboard on the central server to a serial ornetwork attached workstation, in one or more formats specified by thecustomer prior to test setup FIG. 12.

Deficiencies in network performance compared to customer service levelrequirements may invoke the collection of additional information throughboth the mobile wireless network service quality test device 115 andcommunication with third party devices through the central server's API460. All gathered data may then be submitted to the central server'sanalytics engine module where it, data obtained from wireless equipmentmanufacturers and vendors, specialized predictive programming and datastored from past test wireless network runs may be used to formulate aroot cause. Notifications, warnings and alarms specified by the customerwill also be implemented at this point. All data will be saved in datastore 485 for possible further analysis and record keeping.

FIG. 5 is a flow diagram illustrating an exemplary method 500 by whichcentral analysis and control server may interact with other componentsof the system and analyze network performance test data according to apreferred embodiment of the invention. In this embodiment, the centralanalysis and control server 111, 410 first programs a plurality ofmobile network test devices 115 with test suite and connected parameters501. These test may recur automatically at set time intervalsindefinitely, recur a predetermined number of times either regularly orpseudo-randomly, or occur one time only. Mobile network test devicesthen carry out those tests and send the wireless network performancedata back to the centralized analysis and control server 502. Thecentralized server then applies a plurality of customer specifiedpreprogrammed analytical functions on that data 504 and may also employdata obtained directly from third party wireless network equipment suchas, but not limited to: wireless access points and signal amplifiers andrepeaters 503. A major focus of analysis is to determine whether thereare any wireless network performance parameter that fall belowpre-established customer service quality level minimums 505. When allparameters are found to be at or above pre-established customer servicequality level minimums, results are sent to long term storage 508 andthey are presented as per specified customer rendering choices. However,if any of the parameters specified by the customer to have a minimumacceptable service quality level are found to function below that level506 customer required, notifications warnings may be automatically sent,and, based on severity and equipment probed, alarms may be activated andmessages sent to third party equipment monitoring and control systemsthrough the central server's API 112, 460. All results are placed intolong term storage 508.

FIG. 6 is a diagram illustrating information that is processed bycentral analysis and control server analytics engine according to apreferred embodiment of the invention. Within the embodiment 600,central server analytics engine module 445 retrieves and makes use ofmany data from a plurality of sources to predict wireless networkequipment that may be cycling towards failure or after a failure orsignificant deficiency in service has been discovered though testing,steps that may remedy the issue. Information employed comes from a setthat includes the test result database 485, test configuration controls601 and other sources that may include, but are not limited to, WLANmanagement data—tag-specific access points reporting errors in log,non-responsive 602, mobile network management data 603—to inform onwireless phone coverage issues, location service data (whether testdevice is indoors or outdoors)—test devices outside the building mayshow poor service, possibly by design 604, wired network managementdata-issues with wired network service may significantly affect resourceavailability outside of wireless service performance 605, deep packetinspection data—streams of packets from host, or malformed packets fromhost may be easily halted and service quality restored 606, applicationperformance management data—heavily used mobile wireless network testdevice may issue inaccurate performance data 607, application data 608and current alarm and notification status—In times of issue havenotifications been sent, alarms activated 609. As described above rootcause and potential remedies may be offered by the central analyticsengine module 601.

FIG. 7 is a block diagram illustrating an exemplary system architecturefor mobile wireless service quality test device employed to analyzenetwork service quality using remote devices, according to a preferredembodiment of the invention 700. The mobile wireless network servicequality test device of the embodiment incorporates many of thecomponents present in any handheld mobile device such as: antenna 711,RF transceiver and amplifiers 712, speaker 723, CPU and assorted devicecontrollers 713, display controller 714, input controller 715, operatingsystem 717, memory 721, mobile network controller 716—found in allwireless phone and data network connecting devices and an LCD display,not depicted for simplicity. The device operates as a wireless networkperformance probe through use of wireless network service quality testframework application 718 which in itself can run a plurality of simplenetwork performance tests as a standalone device and when programmedwith specific customer wireless network test suites by the system'scentral analysis and control server 450 can run a larger number of testsand send them back to the central server for further analysis, storageand reporting. Tests are run using the networking resources of thedevice through the wireless access circuitry present 711, 712, 713, 716.Results of each test which may include, but are not limited to steadystate signal strength, variability in signal strength, signal to noiseratio, current percentage of network traffic versus network capacity,radio attachment latency, resource request latency, location of the testdevice, and ping response time are collected and transmitted to thecentral analysis server by direction of wireless test data collectionand delivery module 719. The wireless network test device may alsodisconnect from the primary network temporarily to perform such tests aspacket analysis and to scan the radio bandwidths used by the customer'swireless network for interference, multi-path effects. Similarly, thewireless network test device 710 may disconnect from the internalwireless network to measure the signal strength, connection latency anddata rate of the customer's mobile phone provider's voice and dataconnection. Those skilled in the art may quickly recall other tests tobe run, the list provided here is illustrative and should not be thoughtas limiting to the invention. When not connected to the central analysisand control server 410, the mobile wireless network test device uses itsresident network test application data store 720. Once connection to thecentral analysis and control server is reestablished, the wireless testdata collection and delivery module 719 retrieves the stored data fromthe network test application data store 720 and transmits it to thecentral analysis and control server 410. Under certain circumstances,the mobile wireless test device may send information inquiries to knownhigh volume web sites such as, but not limited to FACEBOOK™, SKYPE™,CHROME™ and GOOGLE™ using wireless test application inbound API 722 aspart of a test suite.

FIG. 8 is a flow diagram illustrating an exemplary method by whichmobile wireless network service quality device may operate to collectstore and transmit test suite data according to a preferred embodimentof the invention 800. The mobile wireless network service quality devicereceives coding and parameters for a customer directed networkperformance test suite from the central analysis and coding server 801which it stores 802 and uses in conjunction with a preloaded residenttest framework application 803. The tests of the test suite uploaded maybe executed interactively or programmed to run in the background,possibly in an automatic recurring fashion that continues indefinitelyor is finite in their repetition 803. Single execution tests, are ofcourse, also possible. Occasionally, a test or group of tests causemobile wireless network test device to temporarily disconnect fromprimary wireless network and thus central analysis and control server,the target of collected wireless network performance data. Unexpecteddisruptions in primary network connection can also cause temporary lossof contact with central analysis and control server 804. Under theseconditions wireless performance test data collected by mobile wirelessnetwork test device are stored in a test framework initiated data store805. All wireless network performance data is eventually transmitted tothe central analysis and control server 410 for transformation and longterm storage 806.

FIG. 9 is a diagram illustrating methods by which users of wirelessnetworks being tested might submit their perceived experience to thewireless network service quality test system according to a preferredembodiment of the invention 900. It is often helpful when executingequipment performance tests, wireless network service quality tests inthis case, to also know how a particular service level is perceived bythe equipment users. Correlation between very positive performance testreadouts and a similar user perception of service quality solidifies thevalidity test. Similarly, understanding the effect of what is consideredmediocre, or unacceptable performance test readouts on user perceptionis equally important, if the test results are poor but the user responseis positive at the same time, the tests employed may be inaccurate ormay not be calibrated correctly. The invention may allow the collectionof user experience feedback as part of wireless network service qualitytesting. In the current embodiment of the invention, user wirelessnetwork performance experience is collected using an applet that is partof the wireless network testing framework on the wireless network testdevice 910. A part of the applet displays a multiple choice request foruser wireless network performance experience where the user can ratecurrent network performance from one of five choices: “Excellent”,“Good”, “Sufficient”, “Not Sufficient” and “Very Poor” 911. The secondportion of the applet instructs the user to describe any issues she ishaving on the wireless network in freeform text 912 and then gives spaceto do that 913. Response text is interpreted and mapped to standardizedterms by the central analytics engine as part of overall testing 914.The choice from the first portion 911 of the applet are similarly usedto measure general wireless network performance.

FIG. 10 is a diagram illustrating both passive and active testing ofmore than one wireless network by wireless network service quality testdevices according to a preferred embodiment of the invention 1000. Thesystem 1001 has the ability to run tests on two separate networks withina single test suite. In the case of testing the corporation's wirelessphone provider network connection 1003, this testing may occur actively,attempts to connect to designated phone exchanges, to downloaddesignated data or to connect to designated servers and web pages tolist a few illustrative examples, or passively, measurements of signalstrength at test location, signal to noise ratios, radio and connectacknowledgement latency. Results may be reported to the phone serviceprovider's quality management as well as client corporate analysts onceconnection with the primary wireless network 1004 is reestablished atthe conclusion of mobile network testing 1003. When two separatewireless networks 1004 that require use of the RF radio is planned, thismay be accomplished in two ways. First, the remote test device may havesimultaneous dual RF radio capability, in which case the second wirelessnetwork not depicted for simplicity may be probed and the resultant dataimmediately sent to the central analysis and control server 410 forinclusion in the analysis results made available to the end user 480.Alternatively, a remote test device with only single RF capability maydisconnect from the first network 1004, connect to the second network tobe tested and while probing the network, store the resultant datainternally. Upon completion of inspection of the second network, theremote test device may disconnect from the second network, re-connect tothe first network 1004 ad send all of the stored result data from thesecond network to the central analysis and control server 410 forinclusion in the test results of the current test suite and presentationto the end users 480. A second network service quality test device 1002with both wireless 1004 network and wired network 1005 connectioncapabilities may be used to execute interactive and background networkperformance tests, either active such as data throughput, retry number,and ping return time or passive such as deep packet inspection, packetcapture and protocol analysis, packet drop rates and wireless radiochannel usage is manners similar to those described above. Again, datawould be sent to the central analysis and control server 410 as soon asa connection is reestablished.

FIG. 11 Is a list of methods by which the location of individualwireless network service quality test devices may be obtained accordingto a preferred embodiment of the invention 1100. The ability to positioneach wireless network test device at a specific location within thegeography of the network greatly augments the value of the datacollected as such things as coverage maps and access pointidentification can be made. The GPS system 1101 is an extremelyaccurate, automated method to establish test device location and alsoallows determination of test device movement, which can affect test dataas the orientation of the device antenna changes and altitude which mayplay a role in specific circumstances. GPS signals can be weak orerratic indoors where may wireless network performance tests occur,then, other methods such as location determination by proximity to knownWi-Fi SSID/BSSID or visible light based data transmitter (“Li-Fi”) orinfrared sensor 1102, Wi-Fi location service, RTLS or similar where testdevice location is determined by differential signal levels, packetpropagation delay or signal direction determination through use of adirectional antenna pattern 1103. Test device location may also bedetermine using a magnetic location service which has been calibratedfor a particular building or floor plan 1104. Less reliable, as itrelies on manual response and a user's perception of the preciselocation she finds herself is to have the user identify her location ona floorplan map 1105 or verbally provide her location in an uncoached,freeform manner 1106.

FIG. 12 is a diagram displaying example forms of wireless servicequality test output according to a preferred embodiment of the invention1200. The invention can display both interim and final results ofwireless service quality analysis in a large plurality of ways anddisplay method may be significantly affected by the desires and needs ofindividual customers. Multiple exemplary display methods from one ormore embodiments of the invention are shown in FIG. 12A, and FIG. 12B.The first example, 1201 illustrates the use of line graphs to show oneaspect of wireless network performance, file download from a remotenetwork server to mobile network service quality devices 710 on twoseparate wireless networks 1204, 1205. The x-axis denotes time in hourswith six hour intervals specifically labeled 1203. The y-axis denotesdownload speed in megabits per second. It can be seen that downloadspeed varies significantly over time for both tested network 1 1204 andtested network 2 1205 and variance is cyclic for both networks. Thevariance is highly synchronized between the two networks with valleys ofbandwidth occurring largely concurrently for both network. It shouldalso be noted that for both tested network 1 1204 and network 2 1205,these lows fall below service quality minimum set by the customer forthis activity 1206 where warnings may be sent out. Notifications maythus be sent to administrators, technicians, and managers per thecustomer's problem handling plan. Further, during the test period shown,the performance quality of network 2 1205 falls below the leveldesignated by the customer as “critical” on multiple occasions. Theseevents may lead to more drastic notification of a wider group of peoplewithin the customer company and may also lead to the activation of oneor more alarms to signify critical status as is illustrated in FIG. 13.

A second example shows a pseudo-topographical map which may besuperimposed over the floorplan of the area served by a wirelessnetwork, or may depict individual wireless access point designations asmarkers and which relies on the location of wireless network servicequality devices. This map conveys three parameters important to wirelessnetwork performance: shading represents congestion and retransmissionlevels with darker shading denoting diminishing service quality; Theheight of the graph denotes wireless signal strength; and the linesdenote user reported wireless service experience with denser groupingsindicating diminishing user service quality experience. The pin at 1215denotes a wireless access point and the expected high signal strength isseen 1214. 1213 shows that although the signal strength is somewhatdiminished with distance from the access point 1215 and test measuredcongestion and packet retransmission level appear relatively high, userexperience is good to excellent. Two apparent problem areas for thisnetwork appear to be at 1216 and 1217 where test measured signalstrengths are low, test measured network congestion and packetretransmission levels are high and user reported service qualityexperience is very poor. These results allow the company to take actionif desired. Legend 1212 represents a compass bearing to give an externalreference.

A third example shown in FIG. 12B, 1220 depicts a map of functionalwireless network coverage for a specific customer 1221. This coveragemap is the result of programmatic transformation by the central controland analysis server followed by display in a format that may be specificto or modified for the needs of the customer. This example should not beseen to limit the capabilities of the invention to display of networkparameter data using maps or the complexity that such display may attainbut is meant only to introduce such map use in an easy to visualizemethod.

A fourth example is a set of graphs illustrating test results from asubset of the network parameters from a plurality of possible parameterswhich include ping response time 1230; web page download time 1240;network packet throughput 1260; voice over IP voice quality 1270; TCPdownload throughput 1250; and TCP upload throughput 1280 from a specificnetwork device. All graphs measure service parameter quality as afunction of time 1232, 1242, 1252, 1262, 1272, 1282. The graphs for pingresponse time 1230, web page download time 1240, network packetthroughput 1260 and voice over IP voice quality 1270 report thoseparameters with respect to customer service level agreement (SLA)requirements 1231, 1241, 1261, 1271 with service level plotted as thepercentage of that SLA 1233, 1243, 1263, 1273. Each graph shows apercentage of customer SLA that would lead to warning notifications1234, 1244, 1264, 1274 being sent as illustrated in FIG. 13. Some of thegraphs plot parameters on which the customer relies to have functioningat or very near 100% of their SLA level 1240, 1260 1270 whereas theembodiment is set to warn those designated only after ping responsedrops below approximately 80% of the customer's SLA 1234. All fourgraphs also have a critical service level set 1235, 1245, 1265, 1275.Service levels below these critical set points may cause more extremenotifications to be sent, possibly to a larger group of designatedcustomer representatives and alarms may also be activated 1300. As anexample of such a situation, it can be seen in the ping response timegraph 1230 that service level not only falls below the level wherewarnings may be sent 1234, but falls below 1236 the customer's“critical” service level 1235. This may lead to alarms being activatedat that time, possibly per the process illustrated in FIG. 13. The lasttwo graphs 1250 and 1280 are plotted using time 1252, 1282, but use theless derived measure of Mbits/s for the y-axis 1251, 1281 with theextent of the y-axis fitted closely to the maximal throughput 1253, 1283in both of these situations, warnings 1254, 1284 and criticalnotifications 1255, 1285 are both set such that a network failure wouldbe required, or were not set by the customer. SLA and MBits/s are twoexamples of measurements that may be graphed by the invention but arenot exhaustive. The invention may be used to display any measurementknown to those skilled in the art and appropriate to the parameter beingdisplayed.

FIG. 13 is a flow diagram of a possible notification, warning and alarmmechanism to be used when network performance is found to be belowcustomer established service level minimums according to a preferredembodiment of the invention 1300. Wireless service service quality testresults once analyzed 1301 by the central analysis and control server410, may be compared to customer service level requirements 1302. Oftenall results will be above customer minimums and those results may bedisplay according customer specifications 1303. Otherwise, network widenotifications 1304 will be sent to those representatives designated bythe company according to the customer's notification policy. Messages orreadouts identifying the faulty equipment or suggesting remedial stepsas calculated by the central analytics engine module 445 may also besent or displayed 1305. If there are customer alarms set for thedetermined deficiencies 1306, signals to activate those alarms will besent 1307. Again, messages or readouts identifying the faulty equipmentor suggesting remedial steps as calculated by the central analyticsengine module 445 may be sent of displayed 1308. It is also possiblethat the analyzed test results show that only a specific portion of thewireless network tested is impacted 1309, if the customer has policy forsuch circumstances those will be followed 1310, if not the informationmay be included in any general network notifications. Once again,messages or readouts identifying the faulty equipment or suggestingremedial steps as calculated by the central analytics engine module 445may be sent of displayed. In all cases the test data will be displayedon the interactive terminal 480 and placed in long-term storage 1312.

FIG. 18 is a diagram illustrating a change of access point duringwireless network user roaming, according to an embodiment of theinvention 1800. Roaming within an area of network coverage by a userwith a mobile device along a spatial dimension 1802 may present specialchallenges for network service quality, especially during certainnetwork tasks. The issue may occur at a point where signal strength 1801of a first access point 1804 decreases and signal strength of a second,adjacent access 1803 point increases to the point where the user'snetwork device switches connection from the first access point to thesecond access point at a particular time 1817. As illustrated, thisevent 1817 may be configured such that the signal level from the firstaccess point 1804 will have decreased significantly below 1806 theavailable signal strength 1805 of the second access point 1803, towardswhich the user is traveling, before the wireless device switches accesspoints at time 1817 so as to minimize rapid, repetitive, switches of thedevice from one to the other access point. The point of the roamingevent 1817 at access point coverage boundaries may be mapped, and isaffected by several factors including but not limited to environment(such as weather), mobile or transient obstructions, user devicecharacteristics, and radio interference from other equipment.

The rate of occurrence of roaming events 1817 is minimized as, duringeach event 1817, there is a period of time, usually quite small, wherethe user device is breaking down the connection to the first accesspoint 1807 and establishing the connection with the second access point1809; during this time packets may be lost 1808, incurring an effective“roaming delay”. Often, this event will go unnoticed by the user as thelost packets are retransmitted and the packet stream at the point of theroaming event 1817 reassembled with no noticeable effect. Someapplications, such as (but not limited to) voice over IP, are highlysensitive to packet loss and loss of coherence or stutter may occur atpoints of roaming delay 1808. It is also possible that operation of oneor more of the access points in a customer's network may lead to eitherstatic or transient elongation of roaming delay events 1808, which maylead the customer to implement an embodiment of the invention that teststhis parameter at critical boundaries within the network.

FIG. 19 is a table 1900 of network parameters that may be testedaccording to an embodiment of the invention. There are many networkparameters that are measurable, many of which may be of great importanceto customers reliant on knowing and maintaining the service qualitylevel of their wireless networks at one or more sites, and which can betested and analyzed by embodiments of the invention. These measurableand analyzable parameters range from level 1 1901 of the OSI model tolevel 7 1902 of that model. A non-exhaustive listing of testableparameters at each OSI level are listed in table 1900. These range fromradio spectrum analysis of bands associated with wireless networkfunction and properties of the radio signals at the network site at thephysical level; beaconing, and traffic volume at the network level; andpacket loss and throughput of UDP, TCP and ICMP packets at the transportlevel, just to list a few examples from the already somewhat limitedexample listing 1900. This list 1900 is purely illustrative and theabsence of a parameter from the listed parameters does not at all implythat the invention is incapable of probing its functional level. Theinvention is designed to probe any measurable parameter known to thoseskilled in the art and requested by customer specification.

Hardware Architecture

Generally, the techniques disclosed herein may be implemented onhardware or a combination of software and hardware. For example, theymay be implemented in an operating system kernel, in a separate userprocess, in a library package bound into network applications, on aspecially constructed machine, on an application-specific integratedcircuit (ASIC), or on a network interface card.

Software/hardware hybrid implementations of at least some of theembodiments disclosed herein may be implemented on a programmablenetwork-resident machine (which should be understood to includeintermittently connected network-aware machines) selectively activatedor reconfigured by a computer program stored in memory. Such networkdevices may have multiple network interfaces that may be configured ordesigned to utilize different types of network communication protocols.A general architecture for some of these machines may be describedherein in order to illustrate one or more exemplary means by which agiven unit of functionality may be implemented. According to specificembodiments, at least some of the features or functionalities of thevarious embodiments disclosed herein may be implemented on one or moregeneral-purpose computers associated with one or more networks, such asfor example an end-user computer system, a client computer, a networkserver or other server system, a mobile computing device (e.g., tabletcomputing device, mobile phone, smartphone, laptop, or other appropriatecomputing device), a consumer electronic device, a music player, or anyother suitable electronic device, router, switch, or other suitabledevice, or any combination thereof. In at least some embodiments, atleast some of the features or functionalities of the various embodimentsdisclosed herein may be implemented in one or more virtualized computingenvironments (e.g., network computing clouds, virtual machines hosted onone or more physical computing machines, or other appropriate virtualenvironments).

Referring now to FIG. 14, there is shown a block diagram depicting anexemplary computing device 10 suitable for implementing at least aportion of the features or functionalities disclosed herein. Computingdevice 10 may be, for example, any one of the computing machines listedin the previous paragraph, or indeed any other electronic device capableof executing software- or hardware-based instructions according to oneor more programs stored in memory. Computing device 10 may be configuredto communicate with a plurality of other computing devices, such asclients or servers, over communications networks such as a wide areanetwork a metropolitan area network, a local area network, a wirelessnetwork, the Internet, or any other network, using known protocols forsuch communication, whether wireless or wired.

In one embodiment, computing device 10 includes one or more centralprocessing units (CPU) 12, one or more interfaces 15, and one or morebusses 14 (such as a peripheral component interconnect (PCI) bus). Whenacting under the control of appropriate software or firmware, CPU 12 maybe responsible for implementing specific functions associated with thefunctions of a specifically configured computing device or machine. Forexample, in at least one embodiment, a computing device 10 may beconfigured or designed to function as a server system utilizing CPU 12,local memory 11 and/or remote memory 16, and interface(s) 15. In atleast one embodiment, CPU 12 may be caused to perform one or more of thedifferent types of functions and/or operations under the control ofsoftware modules or components, which for example, may include anoperating system and any appropriate applications software, drivers, andthe like.

CPU 12 may include one or more processors 13 such as, for example, aprocessor from one of the Intel, ARM, Qualcomm, and AMD families ofmicroprocessors. In some embodiments, processors 13 may includespecially designed hardware such as application-specific integratedcircuits (ASICs), electrically erasable programmable read-only memories(EEPROMs), field-programmable gate arrays (FPGAs), and so forth, forcontrolling operations of computing device 10. In a specific embodiment,a local memory 11 (such as non-volatile random access memory (RAM)and/or read-only memory (ROM), including for example one or more levelsof cached memory) may also form part of CPU 12. However, there are manydifferent ways in which memory may be coupled to system 10. Memory 11may be used for a variety of purposes such as, for example, cachingand/or storing data, programming instructions, and the like. It shouldbe further appreciated that CPU 12 may be one of a variety ofsystem-on-a-chip (SOC) type hardware that may include additionalhardware such as memory or graphics processing chips, such as a QUALCOMMSNAPDRAGON™ or SAMSUNG EXYNOS™ CPU as are becoming increasingly commonin the art, such as for use in mobile devices or integrated devices.

As used herein, the term “processor” is not limited merely to thoseintegrated circuits referred to in the art as a processor, a mobileprocessor, or a microprocessor, but broadly refers to a microcontroller,a microcomputer, a programmable logic controller, anapplication-specific integrated circuit, and any other programmablecircuit.

In one embodiment, interfaces 15 are provided as network interface cards(NICs). Generally, NICs control the sending and receiving of datapackets over a computer network; other types of interfaces 15 may forexample support other peripherals used with computing device 10. Amongthe interfaces that may be provided are Ethernet interfaces, frame relayinterfaces, cable interfaces, DSL interfaces, token ring interfaces,graphics interfaces, and the like. In addition, various types ofinterfaces may be provided such as, for example, universal serial bus(USB), Serial, Ethernet, FIREWIRE™, THUNDERBOLT™, PCI, parallel, radiofrequency (RF), BLUETOOTH™, near-field communications (e.g., usingnear-field magnetics), 802.11 (Wi-Fi), frame relay, TCP/IP, ISDN, fastEthernet interfaces, Gigabit Ethernet interfaces, Serial ATA (SATA) orexternal SATA (ESATA) interfaces, high-definition multimedia interface(HDMI), digital visual interface (DVI), analog or digital audiointerfaces, asynchronous transfer mode (ATM) interfaces, high-speedserial interface (HSSI) interfaces, Point of Sale (POS) interfaces,fiber data distributed interfaces (FDDIs), and the like. Generally, suchinterfaces 15 may include physical ports appropriate for communicationwith appropriate media. In some cases, they may also include anindependent processor (such as a dedicated audio or video processor, asis common in the art for high-fidelity A/V hardware interfaces) and, insome instances, volatile and/or non-volatile memory (e.g., RAM).

Although the system shown in FIG. 14 illustrates one specificarchitecture for a computing device 10 for implementing one or more ofthe inventions described herein, it is by no means the only devicearchitecture on which at least a portion of the features and techniquesdescribed herein may be implemented. For example, architectures havingone or any number of processors 13 may be used, and such processors 13may be present in a single device or distributed among any number ofdevices. In one embodiment, a single processor 13 handles communicationsas well as routing computations, while in other embodiments a separatededicated communications processor may be provided. In variousembodiments, different types of features or functionalities may beimplemented in a system according to the invention that includes aclient device (such as a tablet device or smartphone running clientsoftware) and server systems (such as a server system described in moredetail below).

Regardless of network device configuration, the system of the presentinvention may employ one or more memories or memory modules (such as,for example, remote memory block 16 and local memory 11) configured tostore data, program instructions for the general-purpose networkoperations, or other information relating to the functionality of theembodiments described herein (or any combinations of the above). Programinstructions may control execution of or comprise an operating systemand/or one or more applications, for example. Memory 16 or memories 11,16 may also be configured to store data structures, configuration data,encryption data, historical system operations information, or any otherspecific or generic non-program information described herein.

Because such information and program instructions may be employed toimplement one or more systems or methods described herein, at least somenetwork device embodiments may include nontransitory machine-readablestorage media, which, for example, may be configured or designed tostore program instructions, state information, and the like forperforming various operations described herein. Examples of suchnontransitory machine-readable storage media include, but are notlimited to, magnetic media such as hard disks, floppy disks, andmagnetic tape; optical media such as CD-ROM disks; magneto-optical mediasuch as optical disks, and hardware devices that are speciallyconfigured to store and perform program instructions, such as read-onlymemory devices (ROM), flash memory (as is common in mobile devices andintegrated systems), solid state drives (SSD) and “hybrid SSD” storagedrives that may combine physical components of solid state and hard diskdrives in a single hardware device (as are becoming increasingly commonin the art with regard to personal computers), memristor memory, randomaccess memory (RAM), and the like. It should be appreciated that suchstorage means may be integral and non-removable (such as RAM hardwaremodules that may be soldered onto a motherboard or otherwise integratedinto an electronic device), or they may be removable such as swappableflash memory modules (such as “thumb drives” or other removable mediadesigned for rapidly exchanging physical storage devices),“hot-swappable” hard disk drives or solid state drives, removableoptical storage discs, or other such removable media, and that suchintegral and removable storage media may be utilized interchangeably.Examples of program instructions include both object code, such as maybe produced by a compiler, machine code, such as may be produced by anassembler or a linker, byte code, such as may be generated by forexample a JAVA™ compiler and may be executed using a Java virtualmachine or equivalent, or files containing higher level code that may beexecuted by the computer using an interpreter (for example, scriptswritten in Python, Perl, Ruby, Groovy, or any other scripting language).

In some embodiments, systems according to the present invention may beimplemented on a standalone computing system. Referring now to FIG. 15,there is shown a block diagram depicting a typical exemplaryarchitecture of one or more embodiments or components thereof on astandalone computing system. Computing device 20 includes processors 21that may run software that carry out one or more functions orapplications of embodiments of the invention, such as for example aclient application 24. Processors 21 may carry out computinginstructions under control of an operating system 22 such as, forexample, a version of MICROSOFT WINDOWS™ operating system, APPLE OSX™ oriOS™ operating systems, some variety of the Linux operating system,ANDROID™ operating system, or the like. In many cases, one or moreshared services 23 may be operable in system 20, and may be useful forproviding common services to client applications 24. Services 23 may forexample be WINDOWS™ services, user-space common services in a Linuxenvironment, or any other type of common service architecture used withoperating system 21. Input devices 28 may be of any type suitable forreceiving user input, including for example a keyboard, touchscreen,microphone (for example, for voice input), mouse, touchpad, trackball,or any combination thereof. Output devices 27 may be of any typesuitable for providing output to one or more users, whether remote orlocal to system 20, and may include for example one or more screens forvisual output, speakers, printers, or any combination thereof. Memory 25may be random-access memory having any structure and architecture knownin the art, for use by processors 21, for example to run software.Storage devices 26 may be any magnetic, optical, mechanical, memristor,or electrical storage device for storage of data in digital form (suchas those described above, referring to FIG. 14). Examples of storagedevices 26 include flash memory, magnetic hard drive, CD-ROM, and/or thelike.

In some embodiments, systems of the present invention may be implementedon a distributed computing network, such as one having any number ofclients and/or servers. Referring now to FIG. 16, there is shown a blockdiagram depicting an exemplary architecture 30 for implementing at leasta portion of a system according to an embodiment of the invention on adistributed computing network. According to the embodiment, any numberof clients 33 may be provided. Each client 33 may run software forimplementing client-side portions of the present invention; clients maycomprise a system 20 such as that illustrated in FIG. 15. In addition,any number of servers 32 may be provided for handling requests receivedfrom one or more clients 33. Clients 33 and servers 32 may communicatewith one another via one or more electronic networks 31, which may be invarious embodiments any of the Internet, a wide area network, a mobiletelephony network (such as CDMA or GSM cellular networks), a wirelessnetwork (such as Wi-Fi, WiMAX, LTE, and so forth), or a local areanetwork (or indeed any network topology known in the art; the inventiondoes not prefer any one network topology over any other). Networks 31may be implemented using any known network protocols, including forexample wired and/or wireless protocols.

In addition, in some embodiments, servers 32 may call external services37 when needed to obtain additional information, or to refer toadditional data concerning a particular call. Communications withexternal services 37 may take place, for example, via one or morenetworks 31. In various embodiments, external services 37 may compriseweb-enabled services or functionality related to or installed on thehardware device itself. For example, in an embodiment where clientapplications 24 are implemented on a smartphone or other electronicdevice, client applications 24 may obtain information stored in a serversystem 32 in the cloud or on an external service 37 deployed on one ormore of a particular enterprise's or user's premises.

In some embodiments of the invention, clients 33 or servers 32 (or both)may make use of one or more specialized services or appliances that maybe deployed locally or remotely across one or more networks 31. Forexample, one or more databases 34 may be used or referred to by one ormore embodiments of the invention. It should be understood by one havingordinary skill in the art that databases 34 may be arranged in a widevariety of architectures and using a wide variety of data access andmanipulation means. For example, in various embodiments one or moredatabases 34 may comprise a relational database system using astructured query language (SQL), while others may comprise analternative data storage technology such as those referred to in the artas “NoSQL” (for example, HADOOP CASSANDRA™, GOOGLE BIGTABLE™, and soforth). In some embodiments, variant database architectures such ascolumn-oriented databases, in-memory databases, clustered databases,distributed databases, or even flat file data repositories may be usedaccording to the invention. It will be appreciated by one havingordinary skill in the art that any combination of known or futuredatabase technologies may be used as appropriate, unless a specificdatabase technology or a specific arrangement of components is specifiedfor a particular embodiment herein. Moreover, it should be appreciatedthat the term “database” as used herein may refer to a physical databasemachine, a cluster of machines acting as a single database system, or alogical database within an overall database management system. Unless aspecific meaning is specified for a given use of the term “database”, itshould be construed to mean any of these senses of the word, all ofwhich are understood as a plain meaning of the term “database” by thosehaving ordinary skill in the art.

Similarly, most embodiments of the invention may make use of one or moresecurity systems 36 and configuration systems 35. Security andconfiguration management are common information technology (IT) and webfunctions, and some amount of each are generally associated with any ITor web systems. It should be understood by one having ordinary skill inthe art that any configuration or security subsystems known in the artnow or in the future may be used in conjunction with embodiments of theinvention without limitation, unless a specific security 36 orconfiguration system 35 or approach is specifically required by thedescription of any specific embodiment.

FIG. 17 shows an exemplary overview of a computer system 40 as may beused in any of the various locations throughout the system. It isexemplary of any computer that may execute code to process data. Variousmodifications and changes may be made to computer system 40 withoutdeparting from the broader scope of the system and method disclosedherein. Central processor unit (CPU) 41 is connected to bus 42, to whichbus is also connected memory 43, nonvolatile memory 44, display 47,input/output (I/O) unit 48, and network interface card (NIC) 53. I/Ounit 48 may, typically, be connected to keyboard 49, pointing device 50,hard disk 52, and real-time clock 51. NIC 53 connects to network 54,which may be the Internet or a local network, which local network may ormay not have connections to the Internet. Also shown as part of system40 is power supply unit 45 connected, in this example, to a mainalternating current (AC) supply 46. Not shown are batteries that couldbe present, and many other devices and modifications that are well knownbut are not applicable to the specific novel functions of the currentsystem and method disclosed herein. It should be appreciated that someor all components illustrated may be combined, such as in variousintegrated applications, for example Qualcomm or Samsungsystem-on-a-chip (SOC) devices, or whenever it may be appropriate tocombine multiple capabilities or functions into a single hardware device(for instance, in mobile devices such as smartphones, video gameconsoles, in-vehicle computer systems such as navigation or multimediasystems in automobiles, or other integrated hardware devices).

In various embodiments, functionality for implementing systems ormethods of the present invention may be distributed among any number ofclient and/or server components. For example, various software modulesmay be implemented for performing various functions in connection withthe present invention, and such modules may be variously implemented torun on server and/or client components.

The skilled person will be aware of a range of possible modifications ofthe various embodiments described above. Accordingly, the presentinvention is defined by the claims and their equivalents.

What is claimed is:
 1. A method for wireless network performancemeasurement and management using remote devices, comprising the stepsof: testing wireless network performance using a wireless networktesting software application on a mobile wireless device, wherein thetesting is performed in background mode while a user of the mobilewireless device performs other conventional wireless device relatedfunctions; obtaining result data from the testing of wireless networkperformance; transmitting the result data to a central analysis andcontrol server; and analyzing the result data at the central analysisand control server using pre-determined transformations to determine anindicia of wireless network service quality; further comprising the stepof downloading customer specific test customizations, such as correcttest profiles and Service Level Agreement (SLA) thresholds, from acentral server to the mobile wireless device when the user enterspre-existing profile credentials which cause the user to be added to aspecific organization for data aggregation.
 2. The method of claim 1,further comprising the step of downloading customer specific testcustomizations, such as correct test profiles and SLA thresholds, from acentral server to the mobile wireless device when the user enters acode.
 3. The method of claim 1, further comprising the steps ofmonitoring passive user network traffic, observing networkresponsiveness, collecting network access data while the user uses themobile wireless device.
 4. The method of claim 1, further comprising thestep of passively measuring for collection wireless radio signal relatedand network traffic load data.
 5. The method of claim 1, furthercomprising the step of postponing active tests when the user runs taskson the mobile wireless device that result in network usage or CPU loadabove pre-set threshold.
 6. The method of claim 1, further comprisingthe step of correlating at least a subset of test data from individualmobile wireless devices with similar wireless mobile devices in acentral database.
 7. The method of claim 1, further comprising the stepof integrating user feedback concerning wireless network service qualitywithin the wireless network testing software application, within userfeedback collection triggered at discretion of an administrator of thecentral analysis and control server, or by pre-determined SLA violationsdetermined by the mobile wireless device application.
 8. The method ofclaim 1, further comprising the step of layering at least one data setwhich includes location information over a scalable map.
 9. The methodof claim 1, further comprising the step of monitoring mobile wirelessdevice connections and transitions between cells and networks using apre-programmed roaming test.